Skip to content

feat(cache): connect fixed GPU blocks to OrbitKV local DRAM - #4

Merged
feichai0017 merged 1 commit into
mainfrom
feat/orbitkv-local-dram
Nov 15, 2026
Merged

feichai0017 merged 1 commit into
mainfrom
feat/orbitkv-local-dram

Conversation

@feichai0017

@feichai0017 feichai0017 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

This adds the first OrbitKV provider: fixed CUDA KV blocks ↔ a same-host Manager's DRAM tier. It uses the existing native registration, Publish/Restore and lease receipts; the adapter performs no byte copies and adds no inference coordinator. OrbitKV and OrbiTE are unchanged.

Depends on #3 and is stacked on feat/cache-identity-budgets for an isolated review. #1 and #2 have been reviewed and merged. Do not merge this before #3.

The supported layout is deliberately narrow: one full-history group, one block per slab, one GPU shard, complete registered layers. Window/state tails, packed slots, CPU tensors and partial layouts are rejected before native submission. In particular, this does not claim full DeepSeek-V4.1 hybrid-cache integration. SSD/peer profiles remain separate.

The provider bounds operations, metadata and borrowed bytes; records producer readiness on the calling CUDA stream; passes the retained stream to Restore; and returns separate logical and retirement receipts. Only a native done receipt releases charges. Pending/unknown completion or failed unregister retains the client, registration, views, handle and charges and fences new work. No uncertain Save/Restore replay. Explicit close joins the storage worker before unregistering. Required native client methods are checked before registration.

API:

from orbitinfer import CacheIdentity, Engine
from orbitinfer.integrations.orbitkv import OrbitKVCacheStorage

identity = CacheIdentity(model="model", weights="immutable-revision",
                         format="runner-cache-v1", dtype="bfloat16",
                         shard="tp=0/1,pp=0/1", semantics="attention-fingerprint")
# layers: runner-owned contiguous CUDA uint8 [physical_blocks, bytes_per_block]
with OrbitKVCacheStorage(socket, identity=identity, layers=layers,
                        block_tokens=block_tokens, max_pending_bytes=budget) as storage:
    with Engine(runner, prefix_cache=True, prefix_cache_storage=storage,
                prefix_cache_identity=identity, prefix_cache_max_bytes=budget) as engine:
        ...  # Existing submit/step/result path.

The source contract is OrbitKV 93c1c55d795c19cb677f6661763d158a37a059af, especially python/src/client.rs, python/src/local_restore.rs and crates/orbitkv-channel/src/client.rs. The adapter is an optional import; importing OrbitInfer loads neither OrbitKV nor Torch.

Handoff:

  • Commit: a463189; P3 first fixed-block local DRAM adapter, building on P0 in fix(cache): retain execution backing and bound external cache transfers #3.
  • Owners: existing Engine storage boundary, one bounded provider executor, OrbitKV native registration/leases/transfer handles. No new page allocator or inference scheduler; no wire change.
  • Validation: uv run --no-sync python scripts/check.py passed: 311 Rust tests, 2,461 Python tests, 551 explicit device/platform skips; Clippy, formatting, installed CPU wheel (including optional adapter import without loading OrbitKV/Torch) and all model package builds passed. ty check, Ruff and git diff --check pass. All 18 adapter controls passed 50 repetitions (900 executions), with byte/guard, budget, identity, cancel, early-drop, receipt and cleanup-failure checks. The full 59-control combined repetition set also passed 50 rounds (2,950 executions). Final head a463189 has 5/5 CI checks passed: Linux and macOS complete checks plus 59 controls x 50 repetitions each, Clippy and lint/types, and Apple/MLX controls.
  • Installed gate (written, not run): uv run --no-sync python tests/hardware/orbitkv_dram.py --bootstrap-socket /run/orbitkv/orbitkv.sock --metrics-url http://127.0.0.1:9090/metrics --device cuda:0 --output /tmp/orbitinfer-orbitkv-dram.json.
  • Limits: GPU ↔ DRAM is implemented at source level, installed profile unqualified. CPU client/CUDA shims are not real GPU or Manager qualification. Manager must be configured DRAM-only; the installed gate rejects advancing SSD/peer route counters. Full DeepSeek layout, SSD, peers and distributed shards are deferred. Requires the inspected native client API; does not assume every published OrbitKV wheel has it.
  • Net delta against fix(cache): retain execution backing and bound external cache transfers #3: +1,278 lines (+1,294/-16), including 498 test lines, 159 installed-gate lines and the guide.

@feichai0017
feichai0017 force-pushed the feat/cache-identity-budgets branch from 9d482fe to 7fe76c8 Compare November 15, 2026 11:43
@feichai0017
feichai0017 force-pushed the feat/orbitkv-local-dram branch from 777d47f to a463189 Compare November 15, 2026 11:43
@feichai0017
feichai0017 marked this pull request as ready for review November 15, 2026 12:09
Copilot AI balanced review requested due to automatic review settings November 15, 2026 12:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@feichai0017
feichai0017 changed the base branch from feat/cache-identity-budgets to main November 15, 2026 12:57
@feichai0017
feichai0017 merged commit e15b5fc into main Nov 15, 2026
5 checks passed
@feichai0017
feichai0017 deleted the feat/orbitkv-local-dram branch November 16, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants